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Intelligence  analysis,  like  other 
complex  tasks,  demands  consider¬ 
able  expertise.  It  requires 
individuals  who  can  recognize  pat¬ 
terns  in  large  data  sets,  solve 
complex  problems,  and  make  pre¬ 
dictions  about  future  behavior  or 
events.  To  perform  these  tasks  suc¬ 
cessfully,  analysts  must  dedicate  a 
considerable  number  of  years  to 
researching  specific  topics,  pro¬ 
cesses,  and  geographic  regions. 

Paradoxically,  it  is  the  specificity  of 
expertise  that  makes  expert  fore¬ 
casts  unreliable.  While  experts 
outperform  novices  and  machines 
in  pattern  recognition  and  problem 
solving,  expert  predictions  of  future 
behavior  or  events  are  seldom  as 
accurate  as  simple  actuarial  tables. 
In  part,  this  is  due  to  cognitive 
biases  and  processing-time  con¬ 
straints.  In  part,  it  is  due  to  the 
nature  of  expertise  itself  and  the 
process  by  which  one  becomes  an 
expert.1 


Becoming  an  Expert 

Expertise  is  commitment  coupled 
with  creativity.  Specifically,  it  is  the 
commitment  of  time,  energy,  and 
resources  to  a  relatively  narrow 
field  of  study  and  the  creative 


1  More  than  200  individuals  contributed  to 
this  study  The  author  is  indebted  to  the  re¬ 
searchers,  fellows,  and  staff  at  the  Center  for 
the  Study  of  Intelligence,  the  Institute  for  De¬ 
fense  Analyses,  the  National  Military  Intelli¬ 
gence  Association,  Evidence  Based  Research, 
Inc.,  and  ANSER  Inc  Staff  and  students  at  the 
CIA  University,  the  Joint  Military  Intelligence 
College,  the  Naval  Postgraduate  School,  Co¬ 
lumbia  University,  Georgetown  University, 
and  Yale  University  also  contributed  to  the 
project. 


energy  necessary'  to  generate  new 
knowledge  in  that  field.  It  takes  a 
considerable  amount  of  time  and 
regular  exposure  to  a  large  num¬ 
ber  of  cases  to  become  an  expert. 

An  individual  enters  a  field  of  study 
as  a  novice.  The  novice  needs  to 
learn  the  guiding  principles  and 
rules — the  heuristics  and  con¬ 
straints — of  a  given  task  in  order 
to  perform  that  task.  Concurrently, 
the  novice  needs  to  be  exposed  to 
specific  cases,  or  instances,  that  test 
the  boundaries  of  such  heuristics. 
Generally,  a  novice  will  find  a  men¬ 
tor  to  guide  her  through  the 
process  of  acquiring  new  knowl¬ 
edge.  A  fairly  simple  example 
would  be  someone  learning  to  play 
chess.  The  novice  chess  player 
seeks  a  mentor  to  teach  her  the 
object  of  the  game,  the  number  of 
spaces,  the  names  of  the  pieces,  the 
function  of  each  piece,  how  each 
piece  is  moved,  and  the  necessary 
conditions  for  winning  or  losing  the 
game. 

In  time,  and  with  much  practice, 
the  novice  begins  to  recognize  pat¬ 
terns  of  behavior  within  cases  and, 
thus,  becomes  a  journeyman.  With 
more  practice  and  exposure  to 
increasingly  complex  cases,  the 
journeyman  finds  patterns  not  only 
within  cases  but  also  between 
cases.  More  importantly,  the  jour¬ 
neyman  learns  that  these  patterns 
often  repeat  themselves  over  time. 
The  journeyman  still  maintains  reg¬ 
ular  contact  with  a  mentor  to  solve 
specific  problems  and  learn  more 
complex  strategies.  Returning  to 
the  example  of  the  chess  player, 
the  individual  begins  to  learn  pat¬ 
terns  of  opening  moves,  offensive 
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Expert  Teams 


and  defensive  game-playing  strate¬ 
gies,  and  patterns  of  victory  and 
defeat. 

When  a  journeyman  starts  to  make 
and  test  hypotheses  about  future 
behavior  based  on  past  experi¬ 
ences,  she  begins  the  next 
transition  Once  she  creatively  gen¬ 
erates  knowledge,  rather  than 
simply  matching  superficial  pat¬ 
terns,  she  becomes  an  expert  At 
this  point,  she  is  confident  in  her 
knowledge  and  no  longer  needs  a 
mentor  as  a  guide — she  becomes 
responsible  for  her  own  knowl¬ 
edge.  In  the  chess  example,  once  a 
journeyman  begins  competing 
against  experts,  makes  predictions 
based  on  patterns,  and  tests  those 
predictions  against  actual  behavior, 
she  is  generating  new  knowledge 
and  a  deeper  understanding  of  the 
game.  She  is  creating  her  own 
cases  rather  than  relying  on  the 
cases  of  others. 

The  chess  example  is  a  rather  short 
description  of  an  apprenticeship 
model.  Apprenticeship  may  seem 
like  a  restrictive  18th  century  mode 
of  education,  but  it  is  still  a  stan¬ 
dard  method  of  training  for  many 
complex  tasks.  Academic  doctoral 
programs  are  based  on  an  appren¬ 
ticeship  model,  as  are  fields  like 
law,  music,  engineering,  and  medi¬ 
cine.  Graduate  students  enter  fields 
of  study,  find  mentors,  and  begin 
the  long  process  of  becoming  inde¬ 
pendent  experts  and  generating 
new  knowledge  in  their  respective 
domains. 

To  some,  playing  chess  may  appear 
rather  trivial  when  compared,  for 
example,  with  making  medical 
diagnoses,  but  both  are  highly  com¬ 
plex  tasks.  Chess  has  a  well- 
defined  set  of  heuristics,  whereas 
medical  diagnoses  seem  more  open 
ended  and  variable.  In  both 
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Apprenticeship ...  is  still 
a  standard  method  of 
training  for  many 
complex  tasks. 
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instances,  however,  there  are  tens, 
if  not  hundreds,  of  thousands  of 
potential  patterns.  A  research  study 
discovered  that  chess  masters  had 
spent  between  10,000  and  20,000 
hours,  or  more  than  ten  years, 
studying  and  playing  chess.  On 
average,  a  chess  master  stores, 
50,000  different  chess  patterns  in 
long-term  memory. J 

Similarly,  a  diagnostic  radiologist 
spends  eight  years  in  full  time  med¬ 
ical  training — four  years  of  medical 
school  and  four  years  of  resi¬ 
dency — before  she  is  qualified  to 
take  a  national  board  exam  and 
begin  independent  practice,3 
According  to  a  1988  study,  the 
average  diagnostic  radiology  resi¬ 
dent  sees  forty  cases  per  day,  or 
around  12,000  cases  per  year.4  At 
the  end  of  a  residency,  a  diagnos¬ 
tic  radiologist  has  stored,  on 
average.  48,000  cases  in  long  term 
memory. 

Psychologists  and  cognitive  scien¬ 
tists  agree  that  the  time  it  takes  to 
become  an  expert  depends  on  the 
complexity  of  the  task  and  the 
number  of  cases,  or  patterns,  to 
which  an  individual  is  exposed. 

The  more  complex  the  task,  the 


1  W  Chase  and  H  Simon,  “Perception  in 
Chess,"  Cognitive  Psychology.  Vol  4,  1973 
pp  55-81 

■'  American  College  of  Radiology  Personal 
communication.  2002 
‘  A  Lesgold,  H  Rubinson,  P  Feltovich.  R 
Glaser,  D  Klopfer,  and  V  Wang,  "Expertise  in 
a  Complex  Skill  Diagnosing  X-Ray  Pictures, ’ 
M  Chi,  R  Glaser,  and  M  Farr,  eds  ,  The  Na¬ 
ture  of  Expertise  (Hillsdale,  NJ  Lawrence  Er- 
Ihaum  Associates,  1988) 


longer  it  takes  to  build  expertise, 
or,  more  accurately,  the  longer  it 
takes  to  experience  and  store  a 
large  number  of  cases  or  patterns 


The  Power  of  Expertise 

Experts  are  individuals  with  special¬ 
ized  knowledge  suited  to  perform 
the  specific  tasks  for  which  they  are 
trained,  but  that  expertise  does  not 
necessarily  transfer  to  other 
domains,3  A  master  chess  player 
cannot  apply  chess  expertise  in  a 
game  of  poker — although  both 
chess  and  poker  are  games,  a  chess 
master  who  has  never  played  poker 
is  a  novice  poker  player.  Similarly, 
a  biochemist  is  not  qualified  to  per¬ 
form  neurosurgery,  even  though 
both  biochemists  and  neurosur¬ 
geons  study  human  physiology.  In 
other  words,  the  more  complex  a 
task  is,  the  more  specialized  and 
exclusive  is  the  knowledge 
required  to  perform  that  task. 

An  expert  perceives  meaningful 
patterns  in  her  domain  better  than 
non-experts.  Where  a  novice  per¬ 
ceives  random  or  disconnected  data 
points,  an  expert  connects  regular 
patterns  within  and  between  cases 
This  ability  to  identify  patterns  is 
not  an  innate  perceptual  skill; 
rather  it  reflects  the  organization  of 
knowledge  after  exposure  to  and 
experience  with  thousands  of 
cases.6 


,  M  Minsky  unci  S  Paperl,  Artificial  Intelli¬ 
gence  (Eugene,  OR  Oregon  Suite  Sy.Mem  of 
Higher  Education,  1974),  |.  Voss  and  T.  Post, 
“On  the  Solving  of  Ill-Structured  Problems," 

M  Chi,  R  Glaser,  and  M  Farr,  eds  ,  Op  Cit 
f'  O  Akin.  Models  of  Architectural  Knowledge 
(London  Pion.  1980),  D  Egan  and  B 
Schwartz  "Chunking  in  Recall  uf  Symbolic 
Drawings,”  Memory  and  Cognition,  Vol  7, 
1979.  pp  149-158,  K  McKeithen,  J  Reitman, 
H  Rueler.  and  S  Hirtle,  “Knowledge  Organi¬ 
zation  and  Skill  Differences  in  Computer  Pro¬ 
grammers,"  Cognitive  Psychology,  Vol  13, 
1981.  pp  307-325 
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Expert*  have  a  deeper  understand¬ 
ing  of  their  domains  than  novices 
do,  and  utilize  higher-order  princi¬ 
ples  to  solve  problems.7  A  novice, 
for  example,  might  group  objects 
together  by  color  or  size,  whereas 
an  expert  would  group  the  same 
objects  according  to  their  function 
or  utility.  Experts  comprehend  the 
meaning  of  data  and  weigh  vari¬ 
ables  with  different  criteria  within 
their  domains  better  than  novices. 
Experts  recognize  variables  that 
have  the  largest  influence  on  a  par¬ 
ticular  problem  and  focus  their 
attention  on  those  variables. 

Experts  have  better  domain-spe¬ 
cific  short-term  and  long-term 
memory'  than  novices  do/  More¬ 
over.  experts  perform  tasks  in  their 
domains  faster  than  novices  and 
commit  fewer  errors  while  prob¬ 
lem  solving/  Interestingly,  experts 
go  about  solving  problems  differ¬ 
ently  than  novices.  Experts  spend 
more  time  thinking  about  a  prob¬ 
lem  to  fully  understand  it  at  the 
beginning  of  a  task  than  do  nov¬ 
ices,  who  immediately  seek  to  find 
a  solution. 10  Experts  use  their 
knowledge  of  previous  cases  as 
context  for  creating  mental  models 
to  solve  given  problems." 


7  At  Chi,  P  Feltovich,  and  R  Glaser,  "Catego¬ 
rization  and  Representation  of  Physics  Prob¬ 
lems  by  Experts  and  Novices,"  Cognitive 
Science.  Vol,  5,  1981,  pp.  121-125.  M.  Weiser 
anti  J  Sheriz.  "Programming  Problem  Repre¬ 
sentation  in  Novice  and  Expert  Program¬ 
mers.”  Instructional  Journal  of  Man-Machine 
Studies.  Vol  14.  1983,  pp  391-396 
H  W  Chase  and  K  Ericsson,  “Skill  and  Work¬ 
ing  Memory,"  G  Bower,  ed  .  The  Psychology 
of  Learning  and  Motivation  (New  York,  NY 
Academic  Press,  19821. 

1  W.  Chase.  "Spatial  Representations  of  Taxi 
Drivers,"  D.  Rogers  and  j.  Slobada.  eds..  Ac¬ 
quisition  of  Symbolic  Skulls  (New  York,  NY' 
Plenum,  1983). 

lu  J.  Paige  and  H.  Simon.  "Cognition  Processes 
in  Solving  Algebra  Word  Problems,"  B  Klein- 
muntz.  ed  ,  Problem  Solving  (New  York,  NY 
Wiley,  1966) 
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An  actuarial  table  is  as 
good,  or  better,  than  an 
expert  at  making  calls 
about  the  future. 
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Better  tit  self-monitoring  than  nov¬ 
ices,  experts  are  more  aware  of 
instances  where  they  have  commit¬ 
ted  errors  or  failed  to  understand  a 
problem.12  Experts  check  their 
solutions  more  often  than  novices 
and  recognize  when  they  are  miss¬ 
ing  information  necessary  for 
solving  a  problem. 13  Experts  are 
aware  of  the  limits  of  their  domain 
knowledge  and  apply  their 
domain’s  heuristics  to  solve  prob¬ 
lems  that  fall  outside  of  their 
experience  base 


The  Paradox  of  Expertise 

The  strengths  of  expertise  can  also 
be  weaknesses, 1-4  Although  one 
would  expect  experts  to  be  good 
forecasters,  they  are  noc  particu¬ 
larly  good  at  making  predictions 
about  the  future,  Since  the  1930s, 
researchers  have  been  testing  the 


11  J.  Voss  and  T  Post,  "On  the  Solving  of  Ill- 
Structured  Problems,"  M  Chi,  R  Glaser,  and 
M  Farr,  eds  .  Op  Cit 

M  Chi.  R  Glaser,  and  E  Rees,  “Expertise  In 
Problem  Solving,"  R  Sternberg,  ed  .  Advances 
in  the  Psychology  of  Human  Intelligence 
(Hillsdale,  NJ  Lawrence  Erlhaum  Associ¬ 
ates.  1982);  D  Simon  and  H  Simon,  “Individ¬ 
ual  Differences  in  Solving  Physics  Problems,” 
R.  Siegler.  cd,,  Children  s  Thinking.  What  De- 
n'lops 7  (Hillsdale,  NJ-  Lawrence  Erlbaum  As¬ 
sociates,  19781 

■•'J  Larkin,  “The  Role  of  Problem  Representa¬ 
tion  in  Physics,”  D  Gentner  and  A  Stevens, 
eds  ,  Mental  Models  IHillsdale,  NJ  Lawrence 
Erlbaum  Associates,  1983'. 

11  C,  Camerer  anti  E  Johnson,  “The  Process- 
Performance  Paradox  in  Expert  Judgment 
How  Can  Expens  Know  so  Much  and  Predict 
so  Badly 7”  K  Ericsson  and  J  Smith,  eds  ,  To¬ 
ward  a  General  Tlteory  of  Expertise  Prospects 
and  Limit  1  Cambridge.  UK  Cambridge  Uni¬ 
versity  Press,  1991  > 


ability  of  experts  to  make  fore¬ 
casts  15  The  performance  of 
experts  has  been  tested  against 
actuarial  tables  to  determine  if  they 
are  better  at  making  predictions 
than  simple  statistical  models.  Sev¬ 
enty  years  later,  with  more  than 
two  hundred  experiments  in  differ¬ 
ent  domains,  it  is  clear  that  the 
answer  is  no. 16  If  supplied  with  an 
equal  amount  of  data  about  a  par¬ 
ticular  case,  an  actuarial  table  is  as 
good,  or  better,  than  an  expert  at 
making  calls  about  the  future. 

Even  if  an  expert  is  given  more 
specific  case  information  than  is 
available  to  the  statistical  model, 
the  expert  does  not  tend  to  outper¬ 
form  the  actuarial  table.17 

There  are  few  exceptions  to  these 
research  findings,  but  the  excep¬ 
tions  are  informative.  When 
experts  are  given  the  results  of  the 


n  H.  Reichenbach,  Experience  and  Prediction 
(Chicago,  1L  University  of  Chicago  Press, 
1938),  T  Sarbin,  “A  Contribution  to  the  Study 
of  Actuarial  and  Individual  Methods  of  Predic¬ 
tion."  American  Journal  of  Sociology,  Vol  48, 
1943,  pp  593-602 

16  R  Dawes.  D  Faust,  and  P.  Meehl,  "Clinical 
Versus  Actual lal  Judgment,”  Science,  Vol.  243, 
1989,  pp,  1668-1674;  W  Grove  and  P.  Meehl. 
"Comparative  Efficiency  of  Informal  (Subjec¬ 
tive,  Impressionistic.)  and  Formal  (Mechani¬ 
cal,  Algorithmic)  Piedtction  Procedures-  The 
Clinical-Statistical  Controversy,"  Psychology, 
Public  Policy,  and  Law.  Vol,  2,  No  2,  1996. 
pp  293-323 

17  K  Dawes,  “A  Case  Study  of  Graduate  Ad¬ 
missions  Application  of  Three  Principles  of 
Human  Decision  Making,”  American  Psychol¬ 
ogist.  Vol.  26,  1971,  pp.  180-188;  W,  Grove 
and  P  Meehl,  Op.  Cit  (see  footnote  16),  H 
Sacks.  “Promises.  Performance,  and  Princi¬ 
ples  An  empirical  study  of  Parole  Decision¬ 
making  in  Connecticut,"  Connecticut  Law  Re¬ 
view,  Vol  9,  1977,  pp  349-422,  T.  Sarbin,  "A 
Contribution  to  the  Study  of  Actuarial  and  In¬ 
dividual  Methods  of  Prediction,"  American 
Journal  of  Sociology,  1943,  pp  48,  593-602,  J 
Sawyer,  “Measurement  and  Piedtction,  Clini¬ 
cal  and  Statistical,”  Psychological  Bulletin, 

Vol  66.  1966,  pp  178-200,  W.  Schofield  andj. 
Garrard.  "Longitudinal  Study  of  Medical  Stu¬ 
dents  Selected  for  Admission  to  Medical 
School  by  Actuarial  and  Committee  Methods,” 
British  Journal  of  Medical  Education,  Vol  9, 
1975.  pp  86-90 
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actuarial  predictions,  for  example, 
they  tend  to  score  as  well  as  the 
statistical  model  if  they  use  the  sta¬ 
tistical  information  in  making  their 
own  predictions, 18  In  addition,  if 
an  expert  has  privileged  informa¬ 
tion  that  is  not  reflected  in  the 
statistical  table,  she  will  actually 
perform  better  than  the  table.  A 
classic  example  is  the  broken  leg 
argument:  Judge  X  has  gone  to  the 
theater  every  Friday  night  for  the 
past  ten  years.  Based  on  an  actuar¬ 
ial  table,  one  would  predict,  with 
some  certainty,  that  the  |udge 
would  go  to  the  theater  this  Friday 
night.  An  expert  knows,  however, 
that  the  judge  broke  her  leg  Thurs¬ 
day  afternoon  and  is  currently  in 
the  hospital  until  Saturday.  Know¬ 
ing  this  key  variable  allows  the 
expert  to  predict  that  the  judge  will 
not  attend  the  theater  this  Friday 
night, 

Although  this  argument  makes 
sense,  it  is  misleading.  Forecasting 
is  not  simply  a  linear  logical  argu¬ 
ment  but  rather  a  complex, 
interdisciplinary,  dynamic,  and  mul¬ 
tivariate  task.  Cases  are  rare  where 
one  key  variable  is  known  and 
weighed  appropriately  to  deter¬ 
mine  an  outcome.  Generally,  no 
single  static  variable  predicts 
behavior;  rather,  many  dynamic 
variables  interact,  weight  and  value 
change,  and  other  variables  are 
introduced  or  omitted  to  determine 
outcome. 


w  L  Goldberg,  “Simple  Models  or  Simple  Pro¬ 
cesses5  Some  Research  on  Clinical  Judg¬ 
ments,"  A  menccln  Psychologist,  Vot,  23,  1968, 
pp  483-^96,  L  Goldberg,  "Man  versus  Model 
of  Man  A  Rationale,  Plus  some  Evidence,  for 
a  Method  of  Improving  on  Clinical  Inferenc¬ 
es,"  Psychological  Bulletin,  Vol  73,  1970.  pp 
422-432,  D.  Leli  and  S.  Filskov,  "Clmical-Ac- 
tuarial  Detection  of  and  Description  of  Brain 
Impairment  with  the  Wechsler-Bellevue  Form 
I  "Journal  of  Clinical  Psychology.  Vol  37, 
1981,  pp  623-629. 
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Theorists  and  researchers  differ 
when  trying  to  explain  why  experts 
are  less  accurate  forecasters  than 
statistical  models.  Some  have 
argued  that  experts,  like  all 
humans,  are  inconsistent  when 
using  mental  models  to  make  pre¬ 
dictions.  That  is,  the  model  an 
expert  uses  for  predicting  X  in  one 
month  is  different  from  the  model 
used  for  predicting  X  in  a  follow¬ 
ing  month,  although  precisely  the 
same  case  and  same  data  set  are 
used  in  both  instances. 19 


A  number  of  researchers  point  to 
human  biases  to  explain  unreliable 
expert  predictions  During  the  last 
30  years,  researchers  have  catego¬ 
rized,  experimented,  and  theorized 
about  the  cognitive  aspects  of  fore¬ 
casting.-0  Despite  such  efforts,  the 
literature  shows  little  consensus 
regarding  the  causes  or  manifesta¬ 
tions  of  human  bias  Nonetheless, 
there  Is  general  agreement  that  two 
types  of  bias  exist: 


15  J,  Fries,  el  al ,  "Assessment  of  Radiologic 
Progression  in  Rheumatoid  Arthritis  A  Ran¬ 
domized,  Controlled  Tnal.”  Arthritis  Rheum  , 
as  written  by  author  Vol  29,  No  1,  1986,  pp 
1-9 

211 J  Evans,  Bias  m  Hu  man  Reasoning  Cults 
es  and  Consequences  (Hove,  UK  Lawrence 
Erlbauin  Associates,  1989).  R  lleuer.  Psychol¬ 
ogy  of Intelligence  Analysts  (Washington,  DC 
Center  for  the  Study  of  intelligence,  1999),  D 
Kahneman.  P  Slovic,  and  A.  Tversky,  ymr/g- 
ment  Under  Uncertainty  Heuristics  and  Bi¬ 
ases  l Cambridge,  UK-  Cambridge  University 
Press,  1982);  A  Tversky  and  D  Kahneman, 
"The  Belief  in  the  Law  of  Small  Numbers,'" 
Psychological  Bulletin.  Vol  76,  1971.  pp  105- 
110,  A  Tversky  and  D  Kahneman,  “Judgment 
Under  Uncertainty  Heuristics  and  Biases," 
Science.  Vol  185,  1974.  pp,  1124-1131 


•  Pattern  bias — looking  for  evi¬ 
dence  that  confirms  rather  than 
rejects  a  hypothesis  and  inadvert¬ 
ently  filling  in  missing  data  with 
data  from  previous  experiences. 

•  Heuristic  bias- — using  inappropri¬ 
ate  guidelines  or  rules  to  make 
predictions. 

The  very  method  by  which  one 
becomes  an  expert  explains  why 
experts  are  much  better  at  describ¬ 
ing,  explaining,  performing  tasks, 
and  problem-solving  within  their 
domains  than  are  novices,  but,  with 
a  few  exceptions,  are  worse  at  fore¬ 
casting  than  actuarial  tables  based 
on  historical,  statistical  models 

A  given  domain  has  specific  heuris¬ 
tics  for  performing  tasks  and 
solving  problems.  These  rules  are 
a  large  part  of  what  makes  up 
expertise.  In  addition,  experts 
need  to  acquire  and  store  tens  of 
thousands  of  cases  within  their 
domains  in  order  to  recognize  pat¬ 
terns,  generate  and  test  hypotheses, 
and  contribute  to  the  collective 
knowledge  within  their  fields.  In 
other  words,  becoming  an  expert 
requires  a  significant  number  of 
years  of  viewing  the  world  through 
the  lens  of  one  specific  domain.  It 
is  the  specificity  that  gives  the 
expert  the  power  to  recognize  pat¬ 
terns,  perform  tasks,  and  solve 
problems. 

Paradoxically,  it  is  this  same  speci¬ 
ficity  that  is  restrictive,  narrowly 
focusing  the  expert's  attention  on 
one  domain  to  the  exclusion  of 
others.  It  should  come  as  little  sur¬ 
prise,  then,  that  an  expert  would 
have  difficulty  identifying  and 
weighing  variables  in  an  interdisci¬ 
plinary  task  such  as  forecasting  an 
adversary's  intentions. 
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The  Burden  on  Intelligence 
Analysts 

Intelligence  is  an  amalgam  of  a 
number  of  highly  specialized 
domains.  Within  each  of  these 
domains,  a  number  of  experts  are 
tasked  with  assembling,  analyzing, 
assigning  meaning,  and  reporting  on 
data,  the  goals  being  to  describe, 
solve  a  problem,  or  make  a  forecast. 

When  an  expert  encounters  a  case 
outside  her  expertise,  her  options 
are  to  repeat  the  steps  she  initially 
used  to  become  an  expert  in  the 
field.  She  can 

•  Try  to  make  the  new  data  fit  with 
a  pattern  that  she  has  previously 
stored, 

•  Recognize  that  the  case  falls  out¬ 
side  her  expertise  and  turn  to  her 
domain's  heuristics  to  fry  to  give 
meaning  to  the  data; 

•  Acknowledge  that  the  case  still 
does  not  fit  with  her  expertise 
and  reject  the  data  set  as  being  an 
anomaly,  or 

•  Consult  with  other  experts. 

A  datum,  in  and  of  itself,  is  not 
domain  specific.  Imagine  eco¬ 
nomic  data  that  reveal  tliat  a 
country  is  investing  in  technologi¬ 
cal  infrastructure,  chemical 
supplies,  and  research  and  devel¬ 
opment.  An  economist  might 
decide  that  the  data  fit  an  existing 
spentling  pattern  and  integrate 
these  facts  with  prior  knowledge 
about  a  country's  economy.  The 
same  economist  might  decide  that 
this  is  a  new  pattern  that  needs  to 
be  remembered  (or  stored  in  long¬ 
term  memory J  for  some  future  use. 
The  economist  might  decide  that 
the  data  are  outliers  of  no  conse¬ 
quence  and  should  be  ignored.  Or, 
the  economist  might  decide  that  the 
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data  would  be  meaningful  to  a 
chemist  or  biologist  and  therefore 
seek  to  collaborate  with  other  spe¬ 
cialists  who  might  reach  different 
conclusions  regarding  the  data  than 
would  the  economist. 

In  this  example,  the  economist  is 
required  to  use  her  economic  exper¬ 
tise  in  all  but  the  final  option  of 
consulting  with  other  experts,  In 
the  decision  to  collaborate,  the 
economist  is  expected  to  know  that 
what  appears  to  be  new  economic 
data  may  have  value  to  a  chemist  or 
biologist,  domains  with  which  she 
may  have  no  experience.  In  other 
words,  the  economist  is  expected  to 
know  that  an  expert  in  some  other 
field  might  find  meaning  in  data  that 
appear  to  be  economic. 

Three  confounding  variables 

affect  the  economist’s 
decisionmaking: 

*  Processing  time,  or  contest.  This 
does  not  refer  to  the  amount  of 
time  necessary  to  accomplish  a 
task,  but  rather  the  moment  in 
time  during  which  a  task 
occurs — “real  time" — and  the  limi¬ 
tations  that  come  from  being 
close  to  an  event.  The  econo¬ 
mist  doesn't  have  a  prion 
knowledge  that  the  new  data  set 
is  the  critical  data  set  for  some 
future  event.  In  “ real  time."  they 
are  simply  data  to  be  manipu¬ 
lated  It  is  only  in  retrospect,  or 
long-term  memory,  that  the  econ¬ 
omist  can  fit  the  data  into  a  larger 
pattern,  weigh  their  value,  and 
assign  them  meaning 


•  Pattern  bias.  In  this  particular 
example,  the  data  appear  to  be 
economic  and  the  expert  is  an 
economist.  The  data  are,  after  all. 
investment  data.  Given  the  back¬ 
ground  and  training  of  an 
economist,  it  makes  perfect  sense 
to  try  to  manipulate  the  new  data 
within  the  context  of  economics, 
despite  the  fact  that  there  may  be 
other  more  important  angles. 

•  Heuristic  bias.  The  economist 
has  spent  a  career  becoming 
familiar  with  and  using  the  guid¬ 
ing  principles  of  economic 
analysis  and,  at  best,  has  only  a 
vague  familiarity  with  other 
domains  and  their  heuristics.  An 
economist  would  not  necessarily 
know  that  a  chemist  or  biologist 
could  identify  what  substance  is 
being  produced  based  on  the 
types  of  equipment  and  supplies 
that  are  being  purchased. 

This  example  does  not  describe  a 
complex  problem — most  people 
would  recognize  that  the  data  from 
this  case  might  be  of  value  to  other 
domains.  Ir  is  one  isolated  case, 
viewed  retrospectively,  which  could 
potentially  affect  two  other 
domains.  But  what  if  the  econo¬ 
mist  had  to  deal  with  one  hundred 
data  sets  per  day?  Now,  multiply 
those  one  hundred  data  sets  by  the 
number  of  potential  domains  that 
would  be  interested  in  any  given 
economic  data  set  Finally,  put  all 
of  this  in  the  context  of  "real  lime." 
The  economic  expert  is  now 
expected  to  maintain  expertise  in 
economics,  which  is  a  full  time 
endeavor,  while  simultaneously 
acquiring  some  level  of  experience 
in  every  other  domain.  Based  on 
these  expectations,  the  knowledge 
requirements  for  effective  collabo¬ 
ration  quickly  exceed  the 
capabilities  of  the  individual  expert. 


61 


Expert  Teams 


The  expert  is  left  dealing  with  the 
data  through  the  lens  of  her  own 
expertise  She  uses  her  domain 
heuristics  to  incorporate  the  data 
into  an  existing  pattern,  store  the 
data  into  long-term  memory  as  a 
new  pattern,  or  reject  the  data  set 
as  an  outlier  In  each  of  these 
options,  the  data  stop  with  the 
economist  instead  of  being  shared 
with  an  expert  in  some  other 
domain.  The  fact  that  these  data 
are  not  shared  then  becomes  a  criti¬ 
cal  issue  in  cases  of  analytic  error.-1 

In  hindsight,  critics  will  say  that  the 
implications  were  obvious — that  the 
crisis  could  have  been  avoided  if 
the  data  had  liven  passed  to  one 
specific  expert  or  another.  In  “real 
time,”  however,  an  expert  cannot 
know  which  particular  data  sec 
would  have  value  for  an  expert  in 
another  domain. 


The  Pros  and  Cons  of  Teams 

One  obvious  solution  to  the  para¬ 
dox  of  expertise  is  to  assemble  an 
interdisciplinary  team.  Why  not 
simply  make  all  problem  areas  or 
country-specific  data  available  to  a 
team  of  experts  from  a  variety  of 
domains?  This  ought,  at  least,  to 
reduce  the  pattern  and  heuristic 
biases  inherent  in  relying  on  only 
one  domain 

Ignoring  potential  security  issues, 
there  are  practical  problems  with 
this  approach.  First,  each  expert 
would  have  to  sift  through  large 


21  L  Kirkpatrick,  CeipUnns  Without  I'.ws  In¬ 
telligence  Failures  in  World  War II  (London 
MacMillan  Company.  F  ShtcK,  fiv- 

renhibh •  Disasters  ITO)'  Governments  Fail 
(Savage,  MD  Rmvirun  -and  LiHleheld,  1991), 
.[  \Viii7.  17m-  'let  Offensnv  Intelligence  tail- 
lire  m  Urtr  (Ithaca.  NY  Cornell  UniveiMtv 
Piess.  1991).  R  Wohl.stetter,  Pearl  Harbor 
Warning  and  Decision  (Stanford.  CA  Scan- 
ford  University  Press,  1962) 
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data  sets  to  find  data  specific  to  her 
expertise.  This  would  be  inordi¬ 
nately  time  consuming. 

Second,  during  the  act  of  scanning 
large  data  sets,  the  expert  inevita¬ 
bly  would  be  looking  for  data  that 
fit  within  her  area  of  expertise 
Imagine  a  chemist  who  comes 
across  data  that  show  that  a  coun¬ 
try  is  investing  in  technological 
infrastructure,  chemical  supplies, 
and  research  and  development  (the 
same  data  that  the  economist  ana¬ 
lyzed  in  the  previous  example!. 

The  chemist  recognizes  that  these 
are  the  ingredients  necessary  for  a 
nation  to  produce  a  specific  chemi¬ 
cal  agent,  which  could  have  a 
military  application,  or  could  be 
benign.  The  chemist  then  meshes 
the  data  with  an  existing  pattern, 
stores  the  data  as  a  new  pattern,  or 
ignores  the  data  as  an  anomaly. 

The  chemist,  however,  lias  no 
frame  of  reference  regarding  spend¬ 
ing  trends  in  the  country  of  interest 
The  chemist  does  not  know  if  this 
is  an  increase,  a  decrease,  or  a 
static  spending  pattern — answers 
that  the  economist  could  supply 
immediately.  There  is  no  reason 
for  the  chemist  to  know  if  a  coun¬ 
try's  ability  to  produce  this 
chemical  agent  is  a  new  phenome¬ 
non,  Perhaps  the  country'  in 
question  has  been  producing  the 
chemical  agent  for  years  and  these 
data  are  part  of  some  normal  pat¬ 
tern  of  behavior. 

One  hope  is  that  neither  expert 
treats  the  data  set  as  an  anomaly, 
that  both  report  it  as  significant. 


Another  hope  is  that  eacii  expert's 
analysis  of  the  data — -an  increase  in 
spending  and  the  identification  of  a 
specific  chemical  agent — will  come 
together  at  some  point  The  prob¬ 
lem  is  at  what  point?  Presumably, 
someone  will  get  both  of  these 
reports  somewhere  along  the  intel¬ 
ligence  chain  Of  course,  the 
individual  who  gets  these  reports 
may'  not  be  able  to  synthesize  the 
information.  That  person  is  sub¬ 
ject  to  the  same  three  confounding 
variables  described  earlier  pro¬ 
cessing  time,  pattern  bias,  and 
heuristic  bias.  Rather  than  solving 
the  paradox  of  expertise,  the  prob¬ 
lem  has  merely  been  shifted  to 
someone  else  in  the  organization. 

In  order  to  avoid  shifting  the  prob¬ 
lem  from  one  expert  to  another,  an 
actual  collaborative  team  could  he 
built  Why  not  explicitly  pur  the 
economist  and  the  chemist  together 
to  work  on  analyzing  data1  The 
utilitarian  problems  with  lilts  strat¬ 
egy  are  obvious.  Not  all  economic 
problems  are  chemical  and  not  all 
chemical  problems  ate  economic. 
Kach  expert  would  waste  an  inordi¬ 
nate  amount  of  time  Perhaps  one 
case  in  one  hundred  would  be 
applicable  to  both  experts;  during 
the  rest  of  the  day,  the  experts 
would  drift  back  to  their  individual 
domains,  in  parr  because  that  is 
what  they  are  best  at  and  in  part 
lust  to  stay1  busy. 

Closer  to  tile  real  world,  the  same 
example  may  also  have  social,  polit¬ 
ical,  historical,  and  cultural  aspects. 
Despite  an  increase  m  spending  on 
a  specific  chemical  agent,  the  coun¬ 
try'  in  question  may  not  lie 
politically,  culturally',  socially,  histori¬ 
cally,  or  otherwise  inclined  to  use  it 
in  a  threatening  way.  There  may  lie 
social  data — unavailable  to  the 
economist  or  the  chemist- — indicat¬ 
ing  that  the  chemical  agent  will  be 
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used  for  a  benign  purpose  In  order 
for  collaboration  to  work,  each  team 
would  have  to  have  experts  from 
many  domains  working  together  on 
the  same  data  set. 

Successful  teams  have  very  specific 
organizational  and  structural 
requirements.  An  effective  team 
requires  discrete  and  clearly  stated 
goals  that  are  shared  by  each  team 
member.22  Teams  require  interde¬ 
pendence  and  accountability — the 
success  of  each  individual  depends 
on  the  success  of  the  team  as  a 
whole  and  the  individual  success  of 
every  other  team  member.2-1 

Effective  teams  require  cohesion, 
formal  and  informal  communica¬ 
tion,  cooperation,  and  shared 
mental  models,  or  similar  knowl¬ 
edge  structures.2-1  While  cohesion, 
communication,  and  cooperation 
might  be  facilitated  by  specific 
work  practices,  creating  shared 
mental  models,  or  similar  knowl¬ 
edge  structures,  is  not  a  trivial  task. 
Creating  shared  mental  models  may 


—  D  Cartwright  and  A  Zander.  Croup  Dy¬ 
namic?  Research  and  theory  (Mew  York, 

\ Y  Harper  &  Row,  I960.).  P  Fundi.  W  Rich¬ 
ardson,  and  H  Conner.  The  Impact  of  Goal 
Setting  on  Team  Simulation  lixpenence,"  Sim¬ 
ulation  and  Gaming.  Yol  21,  Mo  4,  1990.  pp 
-il  1-422,  J  Harvey  and  C  Boeitgei,  improv¬ 
ing  Communication  within  a  Managerial 
Workgroup.  ’  Journal  of  Applied  Behavioral 
Science,  Vni  7.  !97l,  pp  164-j 7a 
2'  M  Deni. sell,  "rile  Effects  of  Cooperation 
and  Competition  Upon  Gioup  Process."  D 
Cartwright  and  A  Zandei,  cds  ,  Op  Cil .  D 
Johnson  and  R  Johnson,  "Tile  Internal  Dy¬ 
namics  of  Cooperative  Learning  Gioups."  R 
Slavin.  S  Sharan.  S  Kagan,  R  Hertz-Lazarow- 
itz.  C  Webb,  and  R  Schmuck.  eds.,  Learning 
to  Cooperate  Cooperating  to  Learn  (New 
Yoik.  NY  Plenum,  1985).  D  Johnson,  G. 
Maruyama,  R  Johnson,  D  Nelson,  and  L 
Skon,  “Effects  of  Cooperative,  Competitive, 
and  Individualistic  Goal  .Siructvre  on  Achieve¬ 
ment-  A  Meta-Analvsis."  Psychological  Bulle¬ 
tin.  Vol  89.  No  1,  1981,  pp,  47-62,  R,  Slavin. 

Research  on  Cooperative  Learning  Consen¬ 
sus  and  Controversy."  Educational  Leader¬ 
ship,  Vol  47,  No.  4,  1989,  pp  52-55,  R  Slavin. 
Cooperative  Learning  (New  York.  NY  Long¬ 
man.  1983) 
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be  possible  with  an  air  crew  or  a 
tank  crew,  where  an  individual's 
role  is  clearly  identifiable  as  part  of 
a  larger  team  effort — like  landing  a 
plane  or  acquiring  and  firing  on  a 
target.  Creating  shared  mental 
models  in  an  intelligence  team  is 
less  likely,  given  the  vague  nature 
of  the  goals,  the  enormity  of  the 
task,  and  the  diversity  of  individual 
expertise.  Moreover,  the  larger  the 
number  of  team  members,  the 
more  difficult  it  is  to  generate  cohe¬ 
sion.  communication,  anti 
cooperation  Heterogeneity  can 
also  be  a  challenge.  It  has  a  posi- 


24  J  Cannon- Bowers,  E  Sains,  S  Converse, 
“Shared  Mental  Models  in  Expert  Team  Deci¬ 
sion  Making."  N.  Castellan,  ed  ,  Currant  Issues 
111  Individual  and  Group  Decision  Making 
( Hillsdale,  NY  Lawrence  Erlhaum  Associates. 
1985);  L  Coeli  and  J  French,  “Overcoming 
Resistance  to  Change,"  D  Quiwrighi  and  A, 
Zander,  eds  .  Op  Cil .  M  Deulsch,  'The  Ef¬ 
fects  of  Cooperation  and  Competition  Upon 
Group  Process,"  D,  Cartwright  and  A  Zandei, 
eds..  Group  Dynamics.  Research  ami  Theory 
(New  Yoik,  NS"  Harper  it  Row,  19601,  L  Fes- 
nnger.  “Informal  Social  Communication."  D 
Cartwright  and  A,  Zander,  eds  ,  Op,  Cil .  D 
Johnson,  K  Johnson,  A  Ortiz,  and  M  Sianne, 
The  Impact  of  Positive  Goal  and  Resouite 
Interdependence  on  Achievement,  Interac¬ 
tion,  and  Altitudes, "  Journal  oj  General  Psy¬ 
chology.  Vol  1 18,  Mo  -i  1996,  pp  341-347,  fi 
Mullen  and  C  Copper,  "The  Relation  Between 
Gioup  Cohesiveness  and  Performance  An 
integiaiion,"  Psychological  Bulletin,  Vol  115, 
199s,  pp  210-227,  W  NiihofandP  Kommerx. 
"An  Analysts  of  Cooperation  in  Relation  to 
Cognitive  Controversy."  R  Slavin,  S  Sharan, 

S  Kagan,  R  Hertz-l.azarowU/.  C  Webb,  and 
R  Schmuck.  eds  ,  Learning  to  Cooperate,  Co¬ 
operating  to  Learn  (New  York,  NY.  Plenum, 
1995).  J  Orasanu,  “Shaiecl  Mental  Models  and 
Ciew  l’erfoi  mance,"  Paper  piesentetl  ai  the 
3-1  til  annual  meeting  of  the  Human  Factors 
Society.  Oilando,  FL,  1990.  S  Seashore, 
Group  Cohesuvness  in  the  Industrial  Work¬ 
group  (Ann  Arbor,  Michigan,  MI  University 
of  Michigan  Press.  1954) 


tire  effect  on  generating  diverse 
viewpoints  within  a  team,  but 
requires  more  organizational  struc¬ 
ture  than  does  a  homogeneous 
team  25 

Without  specific  processes,  organiz¬ 
ing  principles,  and  operational 
structures,  interdisciplinary  teams 
will  quickly  revert  to  being  just  a 
room  full  of  experts  who  ulti¬ 
mately  drift  back  to  their  previous 
work  patterns.  That  is,  the  experts 
will  not  be  a  team  at  all;  they  will 
be  a  group  of  experts  individually 
working  in  some  general  problem 
space  2I’ 


Looking  to  Technology 

There  are  potential  technological 
alternatives  to  multifaceted  teams. 
An  Electronic  Performance  Support 
System  (EPSS),  for  example,  is  a 
large  database,  coupled  with  expert 
systems,  intelligent  agents,  and 
decision  aids  Applying  such  a  sys¬ 
tem  to  intelligence  problems  might 
be  a  useful  goal.  At  this  point, 
however,  the  notion  of  an  inte¬ 
grated  EPSS  for  large  complex  data 
sets  is  more  theory  than  practice.27 


T  Mills,  ‘Powei  Relations  m  1  hree-Pei vm 
Gioups."  D  Camvnghi  and  A  Zandei,  eds,. 
Op  Cil  .  L  Mol  in.  “Linking  Power  Structure 
and  Power  Use,”  K  Cook,  ed  ,  Social  lix- 
eban^e  Theniy  ( Newbury  Park,  CA,  Sage. 
1987),  V  Nieva,  E  Fleishman,  and  A  Kieck. 
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Ignoring  questions  about  the  tech¬ 
nological  feasibility  of  such  a 
system,  fundamental  epistemologi¬ 
cal  flaw's  present  imposing  hurdles. 
It  is  virtually  inconceivable  that  a 
comprehensive  computational  sys¬ 
tem  could  by-pass  the  three 
confounding  variables  of  expertise 
described  earlier. 

An  EPSS.  or  any  other  computa¬ 
tional  solution,  is  designed, 
programmed,  and  implemented  by 
a  human  expert  from  one  domain: 
computer  science.  Historians  will 
not  design  the  “historical  decision 
aid,”  economists  will  not  program 
the  “economic  intelligent  agent," 
chemists  will  not  create  the  "chemi¬ 
cal  agent  expert  system  “  Software 
engineers  and  computer  scientists 
will  do  all  of  that. 

Computer  scientists  may  consult 
w'ith  various  experts  during  the 
design  phase  of  such  a  system,  but 
w'hen  it  is  time  to  sit  down  and 
write  code,  the  programmer  will 
follow  the  heuristics  of  computer 
science  The  flexibility,  adaptabil¬ 
ity.  complexity,  and  usability  of  the 
computational  system  will  be  dic¬ 
tated  by  the  guidelines  and  ailes  of 
computer  science. In  essence, 
one  w'ould  be  trading  the  heuris¬ 
tics  from  dozens  of  domains  for  the 
rules  that  govern  computer  sci¬ 
ence  This  would  reduce  the 
problem  of  processing  time  by  sim¬ 
plifying  and  linking  data,  and  it 
may  potentially  reduce  pattern  bias. 
But  it  will  not  reduce  heuristic 
bias.-9  If  anything,  it  may  exagger- 


-h  R  Jnlmsion  and  J  Fletcher.  U  Metu-Aiiulvsis 
of  the  Effectiveness  of  Computer-Based  Train¬ 
ing  for  Military  l»s/nic/ion  (Alexandria.  VA1 
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however,  intelligence 
analysis  is  neither 
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highly  constrained. 
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ate  it  by  reducing  all  data  to  a 
binary’  state. 

This  is  not  simply  a  Luddite  reac¬ 
tion  to  technology.  Computational 
systems  have  had  a  remarkable, 
positive  effect  on  processing  time, 
storage,  and  retrieval.  They  have 
also  demonstrated  utility  in  identify¬ 
ing  patterns  within  narrowly 
defined  and  highly  constrained 
domains.  However,  intelligence 
analysis  is  neither  narrowly  defined 
nor  highly  constrained.  Quite  the 
opposite,  it  is  multivariate  and 
highly  complex,  which  is  why  it 
requires  the  expertise  of  so  many 
diverse  fields  of  study  Intelli¬ 
gence  analysis  is  not  something  a 
computational  system  handles  w'ell. 
While  an  EPSS,  or  some  other  form 
of  computational  system,  may  be  a 
useful  tool  for  manipulating  data,  it 
is  not  a  solution  to  the  paradox  of 
expertise. 


Analytic  Methodologists 

Most  domains  have  specialists  who 
study  the  scientific  process  or 
research  methods  of  (heir  disci¬ 
pline.  These  people  are  concerned 
with  the  epistemology  of  their 
domain,  not  just  philosophically  but 
practically.  They  want  to  know 
how  experts  in  their  discipline 


reach  conclusions  or  make  discov¬ 
eries  Rather  than  specializing  in  a 
specific  substantive  topic  within 
their  domain,  these  experts  special¬ 
ize  in  mastering  the  research  and 
analytic  methods  of  their  domain. 

In  the  biological  and  medical  fields, 
these  methodological  specialists  are 
epidemiologists.  In  education  and 
public  policy,  these  specialists  are 
program  evaluators.  In  other  fields, 
they  are  research  methodologists  or 
statisticians,  Despite  the  label,  each 
field  recognizes  that  it  requires 
experts  in  methodology  to  main- 
rain  and  pass  on  the  domain’s 
heuristics  for  problem  solving  and 
making  discoveries. 

The  methodologist’s  focus  is  on 
selecting  and  employing  a  process 
or  processes  to  research  and  ana¬ 
lyze  data.  Specifically,  the 
methodologist  identifies  the 
research  design,  the  methods  for 
choosing  samples,  and  the  tools  for 
data  analyses.  This  specialist 
becomes  an  in-house  consultant  for 
selecting  the  process  by  which  one 
derives  meaning  from  the  data,  rec¬ 
ognizes  patterns,  and  solves 
problems  within  a  domain.  Meth¬ 
odologists  become  organizing 
agents  within  their  field  by  focus¬ 
ing  on  the  heuristics  of  their 
domain  and  validating  the  method 
of  discovery  for  their  discipline. 

The  methodologist  holds  a  unique 
position  within  the  discipline  Orga¬ 
nizing  agents  are  often  called  on  by 
substantive  experts  to  advise  on  a 
variety  of  process  issues  within  their 
field  because  they  have  a  different 
perspective  than  do  the  experts.  On 
any  given  day,  an  epidemiologist, 
for  example,  may  be  asked  to  con¬ 
sult  on  studies  of  the  effects  of 
alcoholism  on  a  community  or  the 
spread  of  a  vims,  or  to  review  a 
double-blind  clinical  trial  of  a  new 
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pharmaceutical  product,  In  each 
case,  the  epidemiologist  is  not  being 
asked  about  the  content  of  the 
study;  rather  he  is  being  asked  to 
comment  on  the  research  methods 
and  data  analysis  techniques  used. 

Well  over  200  analytic  methods, 
most  from  domains  outside  intelli¬ 
gence.  are  available  to  the 
intelligence  analyst;  however,  few 
methods  specific  to  the  domain  of 
intelligence  analysis  exist 30  Intelli¬ 
gence  analysis  lacks  specialists 
whose  professional  training  is  in 
the  process  of  employing  and  uni¬ 
fying  the  analytic  practices  within 
the  field  of  intelligence.  Knowing 
how  to  apply  methods,  select  one 
method  over  another,  weigh  dispar¬ 
ate  variables,  and  synthesize  the 
results  is  left  to  the  individual  intel¬ 
ligence  analysts — the  same  analysts 
whose  expertise  is  confined  to  spe¬ 
cific  substantive  areas  and  their 
own  domains'  heuristics. 

Intelligence  needs  methodologists 
to  help  strengthen  the  domain  of 
analysis.  Such  methodologists  need 
to  specialize  in  the  processes  that 
the  intelligence  domain  holds  to  be 
valid.  In  some  fields,  like  epidemi¬ 
ology  and  program  evaluation, 
methodologists  are  expected  to  be 


x  Exceptions  include,  S,  Fcder,  'FACTIONS 
and  Policon  New  Ways  to  Analyze  Politics," 
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Few  analytic  methods 
specific  to  the  domain  of 
intelligence  analysis  exist. 

experts  in  a  wide  variety  of  quanti¬ 
tative  and  qualitative  methods.  In 
other  fields,  the  methodologists 
may  be  narrowly  focused — a  labo¬ 
ratory-based  experimental 
methodologist,  for  example,  or  stat¬ 
istician.  In  all  cases,  however, 
methodologists  can  only  be  effec¬ 
tive  if  they  are  experts  at  the 
process  of  making  meaning  within 
their  own  disciplines 

In  order  to  overcome  heuristic 
biases,  intelligence  agencies  need 
to  focus  personnel,  resources,  and 
training  on  developing  intelligence 
methodologists.  These  methodolo¬ 
gists  will  act  as  in-house 
consultants  for  analytic  teams,  gen¬ 
erate  new  methods  specific  to 
intelligence  analysis,  modify  and 
improve  existing  methods  of  analy¬ 
sis,  and  increase  the 
professionalization  of  the  disci¬ 
pline  of  intelligence. 


Conclusion 

Intelligence  analysis  uses  a  wide 
variety  of  expertise  to  address  a 
multivariate  and  complex  world. 
Each  expert  uses  his  or  her  own 
heuristics  to  address  a  small  por¬ 
tion  of  that  world.  Intelligence 
professionals  have  the  perception 
that  somehow  all  of  that  disparate 
analysis  will  come  together  at  some 
point,  either  at  the  analytic  team 
level,  through  the  reporting  hierar¬ 
chy,  or  through  some 
computational  aggregation. 

The  intelligence  analyst  is  affected 
by  the  same  confounding  variables 
that  affect  every  other  expert:  pro¬ 
cessing  time,  pattern  bias,  and 
heuristic  bias.  This  is  the  crux  of 


the  paradox  of  expertise.  Domain 
experts  are  needed  for  describing, 
explaining,  and  problem  solving; 
yet,  they  are  not  especially  good  at 
forecasting  because  the  patterns 
they  recognize  are  limited  to  their 
specific  fields  of  study.  They  inevi¬ 
tably  look  at  the  world  through  the 
lens  of  their  own  domain's 
heuristics. 

What  is  needed  to  overcome  the 
paradox  of  expertise  is  a  combined 
approach  that  includes  formal  the¬ 
matic  teams  with  structured 
organizational  principles;  techno¬ 
logical  systems  designed  with 
significant  input  from  domain 
experts;  and  a  cadre  of  analytic 
methodologists.  Intelligence  agen¬ 
cies  continue  to  experiment  with 
the  right  composition,  structure, 
and  organization  of  analytic  teams; 
they  budget  significant  resources 
for  technological  solutions;  but 
comparatively  little  is  being  done  to 
advance  methodological  science. 

Advances  in  methodology  are  pri¬ 
marily  left  to  the  individual 
domains.  But  relying  on  the  sepa¬ 
rate  domains  risks  falling  into  the 
same  paradoxical  trap  that  cur¬ 
rently  exists.  What  is  needed  is  an 
intelligence-centric  approach  to 
methodology,  an  approach  that  will 
include  the  methods  and  proce¬ 
dures  of  many  domains  and  the 
development  of  heuristics  and  tech¬ 
niques  unique  to  intelligence.  In 
short,  intelligence  analysis  needs  its 
own  analytic  heuristics  designed, 
developed,  and  tested  by  profes¬ 
sional  analytic  methodologists. 

This  will  require  using  methodolo¬ 
gists  from  a  variety  of  other 
domains  and  professional  associa¬ 
tions  at  first,  but,  in  time,  the 
discipline  of  analytic  methodology 
will  mature  into  its  own  sub-disci¬ 
pline  with  its  own  measures  of 
validity  and  reliability 
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